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Abstract 

We study direct limits (G, K) = lim (G„, Kn) of Gelfand pairs of the form G„ ~ Nn xi Kn 
with Nn nilpotent, in other words pairs (G„, Kn) for which Gn/Kn is a commutative nilmanifold. 
First, we extend the criterion of (W3j for a direct hmit representation to be multiphcity free. 
Then we study direct hmits G/K = lim Gn/Kn of commutative nilmanifolds and look to see 
when the regular representation of G = limG„ on an appropriate Hilbert space \\mL'^{Gn/ Kn) 
is multiplicity free. One knows that the Nn are commutative or 2-step nilpotent. In many cases 
where the derived algebras [n„,n„] are of bounded dimension we construct G„-equi variant iso- 
metric maps C„ : L?{Gn/ Kn) L?' {G n+i / Kn+i) and prove that the left regular representation 
of G on the Hilbert space L'^{G/K) := \m\{L'^{Gn/ Kn),C,n} is a multiplicity free direct integral 
of irreducible unitary representations. The direct integral and its irreducible constituents are 
described explicitly. One constituent of our argument is an extension of the classical Peter- Weyl 
Theorem to parabolic direct limits of compact groups. 



1 Introduction 

Gelfand pairs {G,K), and the corresponding "commutative" homogeneous spaces G/K, form a 
natural extension of the class of riemannian symmetric spaces. Let G be a locally compact topolog- 
ical group, K a compact subgroup, and M = G/K. Then the following conditions are equivalent; 
see |W2l Theorem 9.8.1]. 

1. {G,K) is a Gelfand pair, i.e. L^{K\G/K) is commutative under convolution. 

2. If g,g' £ G then /x^^^ * fi^^,^ = n^^,^ * /i^^^ (convolution of measures on K\G/K). 

3. Cc{K\G / K) is commutative under convolution. 

4. The measure algebra M(K\G/K) is commutative. 

5. The representation of G on L'^(M) is multiplicity free. 

If G is a connected Lie group one can also add 

6. The algebra of G-invariant differential operators on M is commutative. 

Conditions 1, 2, 3 and 4 depend on compactness of K so that integration on M and K\G/K 
corresponds to integration on G. Condition 5 makes sense as long as K is unimodular in G, and 
condition 6 remains meaningful (and useful) whenever G is a connected Lie group. 
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In this note we look at some cases where G and K are not locally compact, in fact are infinite 
dimensional, and show in those cases that the multiplicity-free condition 5 is satisfied. The main 
results are Theorems I5.1UI and 19.101 We first discuss a multiplicity free criterion which extends 
that of |W3j . We used that criterion in |W3j for a number of direct systems {{Gn, Kn)} where the 
Gn are compact Lie groups and the Kn are closed subgroups. Here we use its extension for direct 
systems {{Gn, Kn)} in which a (closed connected) nilpotent subgroup iV„ of G„ acts transitively on 
Gn/Kn- Then (see [W2\ Chapter 13]) Nn is the nilradical of Gn and Gn is the semidirect product 
group Nn x Kn- Further, the group A^^^ is commutative or 2~step nilpotent so its structure and 
representation theory can be clearly understood in terms of the familiar Heisenberg groups. 

Section [3] pins down the structure of direct limit of Heisenberg group, and Section S] provides 
an infinite dimensional analog of the Peter- Weyl Theorem that turns out to be a necessary tool for 
our multiplicity free arguments. 

In Section[5]we look closely at the cases where A^^^ is the (2n + l)-dimensional Heisenberg group 
Hn = ImC + C" and Kn (viewed as a subgroup of U{n)) acts irreducibly on C". The classification 
of these cases is well known; see Table 15.11 It leads to 16 direct systems, listed in Table 15.21 In 
Theorem 15.101 we see that, for each of these systems, the limit space L'^{G/K) is well defined and 
the action of G on iF'iG/K) is multiplicity free. The methods here are explicit and they guide the 
arguments in the more general cases. 

Our basic arguments go in two steps. First we examine the limit Lp'^N) := lirn L^(A^„) as 
an [N X A^)-module, and then we look at the right action of K on L?'{N) in order to analyze 
Lp'iG/K) := \miL?'{Gn/ Kn) as a left G-module. The first step is carried out in Section[6l Section 
[7] works out some preliminaries for understanding the right action of K. Then in Sections [8] and [9] 
specify the precise requirements for the multiplicity free condition. See Theorems 18.61 and 19. II The 
latter reduces some considerations to the case where Nn is a Heisenberg group. 

Tables [9^61 and [9.151 list the key cases of direct systems of nilpotent commutative pairs not based 
directly on Heisenberg groups. The corresponding multiplicity free results are Theorems 19.101 and 

Finally, Appendix A is a small discussion of formal degree for induced representations, and 
Appendix B indicates how some of our results can be made more explicit using branching rules for 
representations of the Kn- 

Our arguments require the direct system {(G„,-R'„)} to have the property that the dim[nn,n„] 
have an upper bound. It seems probable that this condition is not necessary, but that will require 
a new idea. 

2 Direct Limit Groups and Representations 

We consider direct limit groups G = limG„ and direct limit representations vr = lim7r„ of them. 
This means that 7r„ is a representation of Gn on a vector space Vn-, that the Vn form a direct 
system, and that vr is the representation of G on F = limV^ given by Tr{g)v = TTn{gn)vn whenever 
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n is sufficiently large that Vn ^ V and G„ ^ G send Vn to f and g„ to The formal definition 
amounts to saying that vr is well defined. 

Here we will only consider unitary representations. Thus the Vn all will be Hilbert spaces and 
the inclusions Vn ^ Vn+i will preserve norms (and inner products). 

It is clear that a direct limit of irreducible representations is irreducible, but there are irreducible 
representations of direct limit groups that cannot be formulated as direct limits of irreducible finite 
dimensional representations. This is a combinatoric matter and is discussed extensively in |DPW] . 
Dealing with this matter is the crux of the problem of proving multiplicity-free properties. The 
following definition is closely related to the relevant combinatorics but applies to a somewhat 
simpler situation. 

Definition 2.1 We say that a unitary representation vr of G = limG„ is limit— aligned if it is a 
direct limit lim 7r„ of unitary representations in such a way that (i) each group G„ is of type I, (ii) 
TTn is a continuous direct sum f Qn dvniCn) of mutually disjoint primary representations, and (iii) in 
the corresponding inclusions = / duniCn) ^ I ^U+i di^n+iiCn+i) = K-„+i of representation 
spaces map f„-almost-every primary integrand V^,^ of 1/^,^ into a primary integrand V^„+i of . 

Theorem 2.2 Let vr = lim7r„ be a limit-aligned unitary representation of G = lim Gn ■ Suppose 
that the 7r„ are multiplicity free. Then vr is multiplicity free. In other words the commuting algebra 
of TT is commutative. 

Proof. Let V = lini l^-^ be the representation spaces. Consider the primary decompositions 

V„^ = JV(^^ dun{Cn)- Here Vn is a non-negative measure on the analytic Borel space G^ the Cn 
are mutually inequivalent irreducible unitary representations of G^, and iTn = J Cn dvniCn)- 

Since vr is limit-aligned we have maps bn ■ {Gn, i^n) — > (Gn+i, t'n+i) of measure spaces (modulo 
sets of measure zero) such that, for z^^-almost all ^„ G Gn, K-„ ^ maps into V6^((^„). 

Thus we have direct systems 

of irreducible unitary representation spaces for G = lini Gn ■ Let denote the corresponding direct 
limit Hilbert space and C the (necessarily irreducible) direct limit unitary representation of G on 

^C- 

Write Gn for the support of f„ in the hull-kernel topology. From the considerations just above 
we see that 6„(G„ ) C Gn+i ■ That defines a space G' = limG„ , a measure class z/ = limM„ on 
G' , and a decomposition V = JV(^ du{C,). Since the C, are irreducible, the closed vr(G)-invariant 
subspaces of V are just the Vs = ^c ^^(C) where 5 is a measurable subset of G' . Thus any 
two projections in the commuting algebra of vr commute with each other. Since the commuting 
algebra is a W* algebra, thus generated by projections, it is commutative. We have proved that vr 
is multiplicity free. □ 
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3 Direct Limits of Heisenberg Groups 



In this section we work out the structure and properties of the (2-sided) regular representation of 
the infinite Heisenberg groups. That is the foundation for study of the multiphcity free property 
for direct hmits of commutative nilmanifolds. 

Recall that the (ordinary) Heisenberg group is the group Hn = ImC + C" with composition 
{z, w){z\w') = (z + z' + Im (u) • w'),w + w'). Here w ■ w' refers to the standard positive definite 
hermitian inner product on C" and Im v = ^{v — v) is the imaginary component of a complex 
number v. Thus ImC is both the center and the derived group and iJ„/ImC is a vector group. 

For t nonzero and real, 7Tn,t denotes the irreducible unitary representation of i7„ with central 
character e* : (2;, 0) 1— e*^ (we use the fact that z G ImC is pure imaginary). Then 7r„,t is square 
integrable modulo the center of Hn in the sense that its coefficients fu,v{h) = {u,irn^t{h)v) satisfy 
I/I G L^(//„/ImC). For the appropriate normalization of Haar measure iTn.t has formal degree jij" 
in the sense of the orthogonality relation {fu,v, fu',v')L'2{Hn/imC) = \t\^''{u,u') {v,v'). 

The representation space of 7r„ t is the Fock space 



nn,t = If ■.C'^C holomorphic 



J \f{w)\'^exp{-\t\\w\'^)dX{w) <oo 



where A is Lebesgue measure. The representation is 

[7rt{z,v)f]{w) = e*-±*i-(--/2)-/(«; - v) 

where it is the sign oit/\t\. 

For each multi-index m (mi, ... , m^), mj ^ 0, we have the the monomial u;™ = . . . w^" . 
One computes / w"^w"^' exp{—\t\\w\^)dX{w) to see that it is equal to for m 7^ m', and if m = m 

it is equal to cm! for a positive constant c independent of m. Here m! means ^('^iO- Thus we 
can (and do) normalize the inner product on Tit so that the w[m] := tf'"/\/m! form a complete 
orthonormal set in Hn,t- The corresponding space of matrix coefficients is £n,t = '^n,t'^T~^n,t- is 
spanned by the fi^m-t '■ 9 '-^ (""^[1]) ■^n,t(5)-z[w]). These coefficients belong to the Hilbert space 

L2(//„/ImC;e*) = ^ C | |/| € L^Hn/lmC) and f{z,w) = e-''f{0,w)} 



with inner product (/,/') = /^n f{z,w)f'{z,w)dX{w). 

Since is the formal degree of 7rn,t the orthogonality relations say that the inner product in 
£n,t is given by (/i,m;t, /i',m';t) = l^r" if 1 = 1' and m = m', otherwise. Now the |t|"/^/i,m;t form 
a complete orthonormal set in £n,t, and £n,t consists of the functions ^n,t,(p given by 

(3.1) $n,i,v^(/i) = Yl ^l,mmr^'hm-Ah) 



l,m 



where the numbers <y?i,m(i) satisfy Xli m I 

it)\' < 00 
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The Hilbert space L?'{Hn) is the direct integral J^^£n,t\t\'^ based on the complete orthonor- 
mal sets {|i|"^^/i,m;t} in the £n,t- Thus it consists of all functions '^n,ip given by 



/OO /"OO 
^n,tAh)\tr dt= (y,, ^¥'l,m(t)|tr/'/l,m;t(/l)) \t\ 

-OO J —OO ' 



dt 



such that the functions ip\,m : M ^ C are measurable, X^i m l'/'i,m(*)P < c« for almost all t, and 
Eimlv'lmCOP e ^^1^, l^r*^*)- Notethat 



(3.3) -^-"^ 



J — ( 



The left/right representation of Hn x on £n,t is the exterior tensor product 7r„,j Kl vr* It 
is irreducible. The corresponding representation of Hn x Hn on L'^{Hn) is the left/right regular 
representation IT^ := J^^{T^n,t ^ ^r* j)|t|"(it. (The factor is not relevant to the equivalence class 
of the representation, but it is crucial to expansion of functions.) 

Lemma 3.4 The left/right regular representation n„ of Hn x Hn on Lp'{Hn) is multiplicity free. 

Proof. Any invariant subspace must be invariant under the center ImC x ImC of Hn x Hn, so 
it is of the form Jg£n,t \t\'^ dt where 5 is a measurable subset of M. Now any two projections in 
the commuting algebra of n„ must commute with each other. The commuting algebra is a von 
Neumann algebra, so the projections generate a dense subalgebra. Thus the commuting algebra is 
commutative. □ 

Suppose m ^ n. We view an n-tuple w as an m-tuple by appending m — n zeroes; then 
and w[m\ have the same meaning as functions on and on C". Thus also the coefficient function 
f\,m;t ■ Hn — > C is the restriction of /i,m;t : H^. C. From (j3.2p and (13. 3p we see that the map 

(3.5) Cn : Sn,t ^ £m,t defined by C„(|tr/Vl,m;t) = \tr^'' fl,m;t and Cn(^n,^) = 

is an isometric {Hn x i/„)-equivariant injection of £n.t into £m,t- Specifically, it maps a complete 
orthonormal set in £n,t to an orthonormal set in £m,t- 



Remark 3.6 The adjoint of the isometric injection „ : ^n,(/j ^ |t|("-'")/2<^ of L'^{Hn) into 
L?'{Hm) is orthogonal projection of L'^{Hm) to the image of the injection. It is the scalar multiple 
(by |t|('"~")/2) of restriction of functions on each direct integrand £m^t of L'^{Hm), but clearly the 
scalar varies with m, n and t. <0> 



Now let's consider convergence. Let M„ denote the set of multi-indices m of length n. In order 
that ^n,t G £n,t we needed that ip{t) G L?'{Mn), using counting measure. That gives the function 
||(/9|P(f). Now for '^n,ip to be in Lp'{Hn) as n grows, we need G L^(]R, Ifl'^dt) as n grows. These 
conditions are satisfied whenever only finitely many of the (/^i^m are not identically zero, and the 
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nonzero ones are Schwartz class functions of t. Thus we have a dense subset of Lp'{Hn) that maps 
in a norm-preserving way into L'^{Hm), whenever n, and extends by continuity to give a weh 
defined unitary injection L?'{Hn) — > L?'{Hm)- 

Ah the ingredients in the construction of this injection are {Hn x ff„)-equivariant, so the unitary 
injection L?'{Hn) — > L?'{Hm) is equivariant for the left/right regular representation n„ of Hn x Hn- 

Theorem 3.7 There is a strict direct system {L^{Hn), Cm n} of spaces of the Heisenberg groups, 
whose maps Cm n '■ L'^i^n) L'^{Hm) are {Hn x Hn) -equivariant unitary injections. Let n„ denote 
the left/right regular representation of Hn x Hn on L^(Hn) and let H denote the infinite dimensional 
Heisenberg group lini Hn ■ Then we have a well defined Hilhert space L'^{H) := \\m{Lp'{Hn),Cm,n} 
and a natural unitary representation 11 = limlln of H x H on L^{H). Further, that representation 
n is multiplicity-free. 

Proof. All the assertions except the multiplicity-free assertion have just been proved. Now it 
remains only to prove that 11 = lim n„ is limit-aligned, for then Lemma 13.41 and Theorem 12.21 
complete the proof. That alignment is immediate because the unitary injections L^(i7„) — > L'^{Hm) 
are equivariant for the action of the center of Hn x H^ and that action specifies the direct integrands 
£n,t and £m,t within L'^{Hn) and L'^{Hm)- □ 



4 The Peter— Weyl Theorem for Direct Limits of Compact Groups 

We will apply Theorem [321 to the study of direct limits lmi{{Hn x Kn, Kn)}- There Kn is a compact 
connected group of automorphisms of the Heisenberg group Hn. In order to do that we need to 
extend Theorem 13.71 from the Hn to the semidirect product groups Hn x Kn, and for that we must 
first prove the analog of Theorem 13.71 for the direct systems {Kn}- This analog, the Peter- Weyl 
Theorem for {Kn}, is the subject of this section. As we will see in Section [5] we need only study 
the restricted class of direct systems {Kn} given as follows. 



Definition 4.1 Let {Kn} be a strict direct system of compact connected Lie groups, {{Kn)^} the 
direct system of their complexifications. Suppose that, for each n, 

the semisimple part [(tn)c5 (^n)c] of the reductive algebra (tn)c 

V J 

is the semisimple component of a parabolic subalgebra of [tn+i)c- 

Then we say that the direct systems {Kn} and {{Kn)(^} are parabolic and that lim Kn and 
lhn{Kn)(, are parabolic direct limits. This is a small variation on the definitions of parabolic 
and weakly parabolic direct limits in [Wlj . <^ 

Now let {Kn} be a strict direct system of compact connected Lie groups that is parabolic. 
We recursively construct Cartan subalgebras tn C tn with ti C t2 C • • • C t„ C in+i C . . . and 
(using the parabolic property) simple root systems ^„ = (tn)c) such that each simple 

root for (tn)c is the restriction of exactly one simple root for {tn+i)c- Then we may assume that 
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= {V'n,i> • • • )'0n,p(n)} sucli a Way that each ^pn,j is the (in)c~i'estriction of tpn+i,j and of no 
other element of "^n+i- The corresponding sets H„ = {^n,i, . . . ,Cn,p(n)} of of fundamental highest 
weights satisfy: S,n+i,j is the unique element of H„_|_i whose (t„)j.-restriction is .^„.j, for 1 S j = p{n). 

Write 

f^n,\ for the irreducible representation of Kn with highest weight A„ — 'Y^i^ ^n,jS,n,j 
and J^n,\ for its representation space. Choose highest weight unit vectors Vn^x G J^n,\- Note that 
the Lie algebra inclusion t„ ^ tn+i defines an inclusion hl{tn) ^ ^(tn+i) of enveloping algebras. 
Now A{vn^\) ^ -^(^^n+i,A)i £ ^(^n), defines a K^^equivariant injection .7>; ;s^ ^ .^n+i.Ai and 
that injection is unitary because it sends the unit vector Vn^\ to the unit vector w„+i,a- Thus 
we can simply regard Tn,\ as the subspace of J^n+i,\ generated by the highest weight unit vector 

Vn,\ = Vn+l,X- 

The matrix coefficients of Kn,x are the fx.yik) = {x, Kn,\{k)y), and the Schur Orthogonality 
Relations say that their L'^{Kn) inner product is {fx,y, fx',y') = d^s{''^n.x)^^{x,x'){y,y'). Write 
Cn,x for that space of matrix coefficients. For m ^ n let 

Cm,n '■ fx,y (as a function on Kn) ^ ideg{K.m,x)/ deg{Kn,x))^^'^ fx,y (as a function on Km)- 
Then 

C'Jn n '■ ^n.x " * ^m,x is a {Kn X ir„)-equivariant isometric map. 

The Peter-Weyl Theorem L'^(Kn) = Ylx^n.x assembles the Cn,x -^n+i.A into a {Kn x Kn)- 
equivariant isometric injection ^^'^ : L'^{Kn) L'^{Km)- We now come to the following result, 
corresponding to Theorem 13. 7[ 

Theorem 4.3 (Peter-Weyl Theorem for parabolic direct limit groups.) Let {Kn} be a strict direct 
system of compact connected Lie groups. Suppose that {Kn} is parabolic. Then there is a strict 
direct system {L'^ {Kn) Xm,n} of L'^ spaces, whose maps Cm,n • -^^(-^n) — > L'^{Km) are {Kn x Kn)- 
equivariant unitary injections. Let r„, denote the left/right regular representation of Kn x Kn on 
L'^{Kn) and let K = limi^'„. Then we have a well defined Hilbert space L'^{K) := lim{L^(/C„), Q'^ 
and a natural unitary representation T = lirn r„ of K x K on L'^{K). 

The space L'^{K) is the Hilbert space orthogonal direct sum of its subspaces Cx '■= lim/2n,A (^nd 
the action of K x K on Cx is irreducible with highest weight A. In particular the left/right regular 
representation T is multiplicity-free. 

Proof. As in the Heisenberg group setting, it remains only to prove that T = lini r„ is limit- 
aligned, for then Lemma 13.41 and Theorem 12.21 complete the proof. Evidently L'^{K) contains the 
mutually orthogonal spaces Cx '■= lirn Cn,x- If / ^ ^^(^) is orthogonal to all of them, we interpret 
/ as a function on K, and then f\K„ = for all n, so / = 0. Thus L'^{K) is the (Hilbert space 
closure of the) orthogonal direct sum of the Cx ■= lini Cn,x- As A determines the direct summand 
Cx, now r is limit-aligned. □ 

Remark 4.4 Just as we noted in Remark l3.6l the adjoint to the injection Cm,n '• -^^(-^n) L^{Km) 
is orthogonal projection to the image of that injection, and on each (Xm xi('m)~irreducible summand 
of L'^{Km) it is a scalar multiple of restriction of functions. <^ 
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5 Commutative Spaces for Heisenberg Groups 

In this section we put together the results of Sections [3] and S] to study direct systems Kn)} of 

Gelfand pairs where G„ is the semidirect product x of a Heisenberg group with a compact 
subgroup Kn C Aut(i?„). Then Kn is a closed subgroup of the maximal compact subgroup U{n) G 
Aut(i?„). The concrete results in this section will require that be connected and that its action 
on C" be irreducible. 

The classification goes as follows for the cases where Kn is connected and is irreducible on C". 
Carcano's theorem ([C]; or see |BJR1 Theorem 4.6] or |W2t Theorem 13.2.2]) says that (G„,K„) 
is a Gelfand pair if and only if the representation of {Kn)^: on polynomials on C", is multiplicity 
free. Those groups were classified by Kac p£t Theorem 3] in another context. Benson, Jenkins 
and Ratcliff put it together for a classification of these "irreducible Heisenberg" Gelfand pairs 
{Gn, Kn)- See [BJRl Theorem 4.6]. In Section [9] below we'll look at some cases where Kn need not 
be irreducible on C". 

Kac' list (as formulated in [WH (13.2.5)]) is 



Irreducible connected groups Kn C U (n) multiplicity free on polynomials on C" 




Group Kn 


Group {Kn)c 


Acting on 


Conditions on n 


1 


SU{n) 


SL{n;C) 


C" 


n^2 


2 


U{n) 


GL{n;C) 


C" 


n^l 


3 


Sp{ra) 


Sp{m; C) 


C" 


n = 2m 


4 


C/(l) X Sp{m) 


C* X Sp{m: C) 


C" 


n — 2m 


5 


U{1) X SO{n) 


C* X SO{n:C) 


C" 


n ^ 2 


6 


U{m) 


GL{m;C) 


S'^(C™) 


m ^ 2, n = ^m{m + 1) 


7 


SU{m) 


SL{m;C) 


A^(C"') 


m odd, n = ^m{m — 1) 


8 


U{m) 


GL(to;C) 


A^(C") 


n = ^m{m — 1) 


9 


SU{i) X SU{m) 


SL{e;C) X SL{m;C) 


C^® C™ 


n = £to, £ ^ m 


10 


U{1) X SU{m) 


GL{e;C) X SL{m;C) 


C^® C™ 


n = £m 


11 


U{2) X Sp{m) 


GL{2;C) X Sp{m;C) 


(g) C^™ 


n = 4m 


12 


SU{3) X Sp{m) 


SL{3;C) X Sp{m;C) 




n = 6m 


13 


C/(3) X Sp{m) 


GL(3;C) X Sp{m;C) 


«) C^" 


n ~ 6m 


14 


C/(4) X Sp{A) 


GL{4;C) X Sp{A;C) 




n = 32 


15 


SUim) X 5p(4) 


SL{m;C) x Sp{4;C) 




n = 8m, TO ^ 3 


16 


[/(to) X Sp{A) 


GL{m;C) x Sp{4;C) 


C" ® c« 


n = 8m, TO ^ 3 


17 


U{1) X Spin{7) 


C* X Spin{7;C) 


c« 


n = 8 


18 


U{1) X Spin{^) 


C* X Spin{9;C) 




n = 16 


19 


SpinilQ) 


Spin{lQ; C) 




n = 16 


20 


U{1) X SpinilQ) 


C* X Spin{10;C) 




n = 16 


21 


U{1) X G2 


C* X G2.C 




n = 7 


22 


U{1) X Ee 


C* X Ee,c 




n = 27 
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Now we have the direct systems 



Direct systems {{Hn x Kn,Kn)} of Gelfand pairs, 
Kn connected and irreducible on C" 




Group Kn 


Acting on 


Conditions on 77 


1 


SUin) 


C" 


71 ^ 2 


2 


U{n) 


C" 


71 ^ 1 


3 


Sp{m) 


C" 


71 = 27n. 


4 


U{1) X Sp{m) 


C" 


71 = 2m 


5a 


U{1) X 50(2771) 




71 = 2m ^ 2 


5b 


C/(l) X £'0(2771+ 1) 




?i — 2m +1^3 


6 


U{m) 




m ^ 2, n = ^m{m + 1) 


7 


SU{m) 


A^(C") 


TO odd, n — ^m{m — 1) 


8 


U{m) 


A^(C") 


n = ^m(m — 1) 


9 


SU{£) X SU{m) 




77 = £m, £ ^ m 


10 


S{U{£) X U{m)) 




n = £m 


11 


U{2) X Sp{m) 


«) C^™ 


71 — Am 


12 


SU{3) X S'p(77l) 


«) C^" 


n = 6m, 


13 


C/(3) X S'p(77l) 


«) C^" 


n — 6m, 


15 


SU{m) X 5*^(4) 


C™ ® C** 


n ~ 8m, m ^ 3 


16 


;7(?7i) X Sp{4:) 




n — 8m, m ^ 3 



In each case the direct system {Kn} is both strict and parabohc. (We separated entry 5 of Table 
15.11 into entries 5a and 5b of Table [5^21 in order to have the parabolic property.) 



We now suppose that {Kn} is one of the strict parabolic direct system, for example one of the 16 
systems given by the rows of Table [521 We retain the notation of Section [J] for the representations, 
highest weights, unitary inclusions, etc., associated to {Kn}- 

As U{n) acts on Hn = ImC + C" by k : {z,v) ^ {z,kv) it preserves the equivalence class of 
each of the square integrable representations TTn^t of Hn- The Mackey obstruction vanishes and TTn^t 
extends to a unitary representation tt^ of Hn x U{n) on 7in,t- See |WOl Section 4] for a geometric 
proof. We will also write 7t>^ for its restriction, the extension of ■Kn,t to a unitary representation of 

Gn = Hn XI Kn- 

Denote 7rn,t,\ = ® k^x and write 'Hn,t,\ for its representation space 'Hn,t ® ^n,\- Fix an 
orthonormal basis {ui} of J^n,x- Then {t(;[m] ® Uj} is a complete orthonormal set in TCn,t,x- Denote 
the matrix coefficients by 

h,raA,j;t{h, k) = {{w[\] Ui), {{TT^t i^\){h, k)){w[ra] (g) Uj))- 

The formal degree deg7r„ t^;^ = \t\^ deg{Kn,\) , so the deg(K„^A)^^^/i,m,i,j;t form a complete or- 
thonormal set in the space £n,t,\ = 'H.n,t,\®T~^n t x matrix coefficient functions. Given a coefficient 
set If = {fi,m,i,j{t,^)) we have the functions ^n,t,\,<fi on Gn given by 

(5.3) ^n,t,xA^k)= ^ (/7i,m,M(i,A)|tr/2deg(K„;,)V2jj^^ .^^,^(/,^/,)^ heHn,keKn- 

l,m,i,j 



9 



Here \\^n,t,\,v\\l^ ^ ^ = Z]i,m,i j l'^i,m,j,j(i, A)|^. We sum the ^n,t,\,y= form functions ^n,<p on 
Hn>i Kn, given by 

(5.4) ^^l^^^{hk)= degK,,A(^y" ^n,t,x,Ah,f^)\trdty 

As before, for m ^ n we have a x G„)-equivariant isometric injection of L^{G„) into Lp'{Gm) 
given by 

(5-5) C,m,n{^n,<f) = ^m,|t|("-'")/2(deg deg k^,;^)1/2<^ • 

This gives us 

Theorem 5.6 For n > let K„ be a compact connected subgroup of Aut{H„) such that {K„} 
is a strict parabolic direct system. Define Gn = Hn x Kn. Then there is a strict direct system 
{L?'{Gn)Xr,n\ of spaccs, whosc maps C,m,n '■ L'^{Gn) —>■ L'^{Gm) are {Gn X Gn)~equivariant 
unitary injections. Let n„ denote the left/right regular representation of Gn x Gn on L'^{Gn)- Note 
that G := lim(Gn) = H yt K where H = lim Hn and K = hni Kn ■ Thus we have a well defined 
Hilbert space L'^{G) := lim{L^(G„), Cm,n} o,nd a natural unitary representation U = limn„ of 
G X G on L?'{G). Further, 11 is the multiplicity-free direct integral of the irreducible representations 
■Kt,\ := hni 7rn,t.A- 

Now suppose that {Kn} is in fact one of the 16 systems of Table [5^21 We wiU use its specific 
properties in order to pass from the left/right representation of G„ x G„ on L^(G„) to the left 
regular representation of Gn on Lp'{Gn/ Kn). 

Define Gn = Hn x Kn- Since {G^ Kn) is a Gelfand pair with Kn irreducible on C", Carcano's 
Theorem fC] says that the action of Kn on the polynomial ring C[C"] is multiplicity free, and it 
picks out the right i^r„-invariants in Li^iGn)-, as follows. 



Lemma 5.7 Let Kn,x S Kn- Define Kn,\ G Gn by tin,\ 

(hk) 

— ^n,A(^) for h G Hn and k G Kn- Then 
T^n,t,\ '■= ® Kn3y has a nonzcro Kn-fixed vector if and only if ^ occurs as a subrepresentation 
ofT^tlXn, o,nd in that case the space of Kn.- fixed vectors has dimension 1. 

Proof. This is essentially the argument in |WOl Section 14. 5A]. Decompose TUi^t\K„ = E^g/^ "^7 7- 
Carcano's Theorem ([C], or see |W2l Theorem 13.2.2]) says that each is either or 1. The Kn- 
fixed vectors of k vr^ all occur \n {m^*K,*), and they form a space of dimension m^*. The 
assertion follows. □ 

Since Kn is compact, we can view L'^{Gn/Kn) as the space of right --fr„-invariant functions in 
L'^{Gn)- With Lemma 15.71 in mind we set 

Kn = {>^n,\ S Kn \ Kn A occurs in the space of polynomials on C"}. 

Given k„^a £ Kn the right ^C„-invariant in C[C"'](^!F* ^^ is the sum over a basis of the K„^A^subspace 
of C[C"'] times the dual basis of J^* x- Normalize it to a unit vector Un,t,x Then the (left regular) 
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representation of Gn on Lp'{Gn/ Kn) is - — -t f^oo'^'^^^ ® '^'^^^^ representation space is 

n 

.T^t {nn,t,\ ® Un,t,\C) dt. 

Proposition 5.8 If m ^ n and Kn,x € kJ then Km,x £ Krn , and Kn,\ and Km,x have the same 
highest weight A space. 

Proof. The group Kn acts on C"^ by some representation 7„, so the representation of Kn on 
polynomials of degree d is the symmetric power S'^{'j*). Thus we can compute the set Xn,d of 
highest weights of on the space Pn^d of polynomials of degree d on C". Running through the 
16 cases of Table [5^ we see that Xn.d C Xm,d- 

Now let A G Xn,d- Let Vn,\ denote a (nonzero) highest weight A vector for t„ in Pn,d, and 
similarly let Vm,\ denote a (nonzero) highest weight A vector for tm- Divide up the variables of C™ to 
{wi, . . . , Wn} for C"' and {zn+i-, • • • , Zm} for its complement in C". Express Vm,x = b bA,BW^z^ 
where each term has total degree |j4| + \B\ = d. Note that Kn treats the Zi as constants. Evaluating 
the Zi at arbitrary constant values C = (c„+i, . . . ,Cm) we have a highest weight A vector for 
By Carcano's Theorem it is a multiple of f„,A- In other words Vm,x\{z=c} = "ic^",A- The terms 
bA,BW^z^ with z-degree \B\ > yield evaluations of w-degree \A\ < d, and cannot contribute 
to any m^Vn^x- Now bA,BW^z^ = whenever \B\ > 0. This shows that Vm,x is a homogeneous 
polynomial of degree d in the Wj, as is Vn,x- We conclude that Vm,x is a nonzero multiple of Vn^x- D 

Corollary 5.9 Let m ^ n. Then every Kn-invariant vector in 7in,t,x is the image of a K^- 
invariant vector in 'Hm,t,x under the adjoint of the unitary map 7in,t,x ^ T~^m,t,x- 

Proof. Retain the notation Xn,d for those A such that t^^x occurs on the space Pn,d of polyno- 
mials of degree d on C". If A ^ ^n,d there are no nonzero i^„-invariant vectors in 'Hn,t,Xj so the 
assertion is vacuous. Now assume A G Xn^d and choose an orthonormal basis {xi, . . . , Xg^n)} of 
the representation space for Tn^x in Pn,d- According to Proposition 15.81 that representation space 
is contained in the representation space for Tm,x in Pm,d, so the latter has an orthonormal basis 

{xi , . . . , , Xq(„)+i , . . . , } • Let {xj , . . . , X*^n) J" ^'"''^ {^1 ' ' " " ' ^g(ra) ' ^q{n)+l ' ' " " ' ^g(m) J" 

corresponding dual bases of Cn,x and Cm,x- the ir„-invariant vectors in 'Hn,t,x are the multiples 
of Y^'f'^^ Xi® x*^ and the iCm-invariant vectors in Hm,t,x are the multiples of ^^^'"^ Xi x*. The 
adjoint of unitary inclusion is orthogonal projection, which sends Xi x* to Ylf''^^ (g) x*. 

□ 

Combining Theorem 15.61 Lemma 15.71 and Corollary 15.91 we have 

Theorem 5.10 Let {{Hn x LCn,LCn)} be one of the 16 direct systems of Table 15. 2[ Denote Gn = 
Hn XI Kn, G = \miGn and K = limi^^n,. Then the unitary direct system {i^^(G„), Cm,n} o/ The- 
orem [5i6] resiricis to a unitary direct system {L'^{Gn/Kn),Cm,n}, the Hilbert space L'^{G/K) : = 
lim { L'^(Gn/Kn), Cm, n} is the subspace of L'^{G) :=limL^(G„) of right-K -invariant functions, and 
the natural unitary representation of G on L'^{G/K) is a multiplicity free direct integral of lim- 
irreducible representations. 
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Remark 5.11 As in Remarks 13.61 and 14.41 if m ^ n then the adjoint of the direct system map 
(m,n '■ L'^{Gn/ Kn) L?'{Gm/ Km) is orthogonal projection to the image subspace, and on each 
{Gm X Gm)~ii'i'educible direct integrand it is a scalar multiple of restriction of functions. The scalar 
is given by the formal degree, so it depends on the integrand. <) 



6 Extension to Certain Classes of Nilpotent Groups 

Theorem 13.71 depends on four basic facts. First, the T^n,t are determined by their central character. 
Second, we have good models 'Hn,t for the representation spaces, such that n does not appear 
explicitly in the formulae for the actions of the group elements. Third, the injections Hn ^ Hm 
restrict to isomorphisms Z„ = Z^n of the centers. And fourth, we have complete information on 
the Plancherel measure for the Hn- In this section we consider a somewhat larger class of nilpotent 
direct systems that satisfy these conditions. 

We will need the theory of square integrable representations of nilpotent Lie groups ( |M W| . or 
see |W21 Section 14.2] for a short exposition). Let be a connected, simply connected nilpotent 
Lie group and n its Lie algebra. Decompose n = 3 + D and A^ = Zexp(d) where 3 is the center of 
n. Then Z = exp(3) is the center of A^. We say that an irreducible unitary representation vr of 
A^ is square integrable if its coefficient functions fu,v{9) = {u,TT{g)v) satisfy 1/^,^1 G L'^{N/Z). In 
that case vr is determined by its central character, tt = irt where t S 3* and the central character 
is exp(C) 1-^ e**^^-*. In terms of geometric quantization, irt corresponds to the coadjoint orbit in n* 
consisting of all linear functionals on n whose restriction to 3 is t. Further, the antisymmetric bilinear 
form bt{^,r]) = t{[S,,r]]) on is nondegenerate, and (up to a positive constant that depends only on 
normalizations of Haar measures) the formal degree of vr^ is |Pf(6j)|, where Pf(fet) is the PfaffiaE0 
of 6f : D X D ^ M. In fact, if vr^ is the representation of A^ that corresponds to Ad*(A^)s C n*, then 
■Kg is square integrable if and only if Pf(65|^,) 7^ 0. In any case, Pf(6t) is a polynomial function of 
t, and (again up to a constant that depends on normalizations) |Pf(6j)| is the Plancherel density. 
It follows that if one irreducible unitary representation of A^ is square integrable then Plancherel- 
almost-all are. In the case of the Heisenberg group Hn, where we identified 3* with R, the Pfaffian 
corresponding to ■7Tn,t is i"". 

The point of this, from the viewpoint of commutative spaces, is that many Gelfand pairs are of 
the form (A^ x K, K) where A^ is a connected simply connected Lie group, AT is a compact subgroup 
of Aut(A^), and A^ has square integrable representations. See |W21 Theorem 14.4.3]. In quite a 
few cases the groups A^ of [W2l Theorem 14.4.3] fall naturally into direct systems for which we 
can apply the techniques of Section [3l This is simplified by the 2-step Nilpotent Theorem \SM2\ 
Theorem 13.1.1] of Benson-Jenkins-Ratcliff and Vinberg, which says that A^ must be abelian or 
2-step nilpotent. In a certain sense representations treat those groups as Heisenberg groups: 

Lemma 6.1 ( |W2t Lemma 14.4.1]) Let N he a connected simply connected 2~step nilpotent Lie 
group with 1-dimensional center. Then N is isomorphic to the Heisenberg group Hn where n = 
^(dim^ n — 1), and in particular N has square integrable representations. 



^Strictly speaking, Pf(bt) depends on a choice of basis of 0, for a basis change of determinant at multiphes det bt 
by det at and multiphes Pf(&t) by det at. 
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Proposition 6.2 ( |W21 Proposition 14.4.2]) Let N he a connected simply connected 2-step nilpo- 
tent Lie group. Let / S n* such that ^ 0. Denote ro/ = {z G 3 | f{z) = 0} and Wf := exp(lD/). 
Then 

1. Wf is a closed subgroup of Z, hence a closed normal subgroup of N. 

2. The functional f is the pullback of a linear functional f G [n/xof)* and is nonzero on the 
central subalgebra 3/tD/ o/n/toj. 

3. The representation [ttj] is the pullback to N of the class [vrj/] G N/Wj. 

4. If the representation [vrj] is square integrable then [ttj/] is square integrable, and in that case 
N/Wf has center Z/Wf and is isomorphic to a Heisenberg group Hn where n = ^dim(n/3). 

We now consider a strict direct system {Nn} of 2-step nilpotent connected, simply connected 
Lie groups that have square integrable representations, where the inclusions n„, — > rim map the 
center 3„ ^ 3^ and the complement i3„ dm in decompositions nn = In + ^n- Then the direct 
limit algebra n := limrin has center 3 := lim3„ and n = 3 + where = lim \3n- On the group level, 
Z = lirn Zn = exp(3) is the center of N := lirn Nn and we have N = Zexp(d). 

We further assume that the dimensions dim Zn of the centers are bounded. Since they are 
non-decreasing we may assume that they are eventually constant. Passing to a cofinal sequence, 

(6.3) n„ ^ Wm maps 3„ = 3^- 

Under that identification we write 3 for all the 3„, 3* for all the 3* , and Z for all the Zn- 

Let t G 3*. Write bn,t for the bilinear form (^,7?) ^ tii^.iV]) on 0„. Then t corresponds to a 
square integrable representation TTn,t of Nn just when the Pfaffian Pf(6„^t) 7^ 0. For purposes of 
comparing the Pfaffians as n varies, we note that Pf(6n,i) is specified by t and a basis of D„, so 
we simply assume that these bases are nested in the sense that the basis of consists the basis 
of On together with some elements that are 6n+i,t~orthogonal to D„. Thus, if Pf(6„-|-i^t) 7^ then 
Pf (^n,t) 7^ 0. The converse fails in general, but the following lemma deals with the possibility that 
Ff{bn,t) / = Pf(6m,t)- It depends on the fact [MW] that each Ff{bn,t) is a polynomial function 
on 3*. 

Lemma 6.4 Let o„ G 3* denote the zero set of Vi{bn,t) cLnd set a = U"^"- Then a is a set of 
Lebesgue measure zero in 3*. 

Proof. Since Nn has square integrable representations, the Pfaffian Pf(6n,i) is a nontrivial poly- 
nomial function of t G 3*, so a„ is a finite union of lower-dimensional subvarieties of 3* . Now the 
set a is a countable union of sets a„ of Lebesgue measure zero. □ 

For convenience we define 
(6.5) r = {tG3*| each Pf (6„,t) / 0} = 3* \ 0. 

By construction, for every t £ T and every index n we have a square integrable representation 
vr„,t G Nn- 
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Fix t £ T. Then we have the hyperplane tvt in 3, and Wt := exp(lDt) is a closed subgroup of 
Z. Lemma l6.1l and Proposition 16.21 tell us that each quotient N^/Wt is isomorphic to a Heisenberg 
group and that vr^^f factors through to the square integrable representation of N^/Wt with 

central character e**. Now the various (as t varies in T) Trn,t act on the same Fock space 'H(i{n),t> 
d{n) = i dimO„, by formulae independent of d{n). 

We normalize the inner products on the 'Hd{n),t as before, so the w[m\ form a complete or- 
thonormal set, and realize the space £n,t = '^d{n),t®'^*d(n) t matrix coefficients as the span of 
the /i,m,t : g ^ {w\\\,Tin,t{g)w[™^) as in Section [31 The orthogonality relations say that the inner 
product on £n^t is given by (/i,m,ti /i',m',t) = |Pf(fen,t)|~^ if 1 = 1' and m = m', and is otherwise. 
Now the |Pf(6ra,t)|"^^^/i,m,t form a complete orthonormal set in £n.t) and as before £n,t consists of 
the functions ^n,t,ip on -ffrf(„) given by 

(6.6) ^n,tAh) = Yl '^l,m(t)|Pf(&n,t)|'/'/l,m;t(/i) with ^ | 

l,m l,m 

Now L'^(Nn) is the direct integral J^, £n.t\P^{bn^t)\dt = Jrp £n.t\P^{bn,t)\dt. It consists of all functions 
'^n,ip defined by 

(6.7) ^nAh) = j^^ ^n,tAh)mKt)\dt = ( JPX,m{tW^{Kt)\"^ h,ra,^ |Pf(fen,t)|dt 

such that the functions '■ ^^^^ measurable with ^ \ '^\m{t)\^ < 00 for almost all t G T 

and Ei,m W\,m{t)\'^ G L^iln^ \Pi{hn,t)\dt). The norms are 

= / ||$n,i,^|||„JPf(&„,t)|dt 

(6.8) ■'J 

As in the Heisenberg group case, the left/right representation of Nn x Nn on £n^t is the exterior 
tensor product Tr^^t Kl tt* it is irreducible and the left/right regular representation of Nn x Nn 
on L'^{Nn) is n„ = J^,{TTn,t ^ '^n,t)\^Hbn,t)\dt. The argument of Lemma [33] goes through without 
change, proving 

Lemma 6.9 The left/right regular representation n„ of Nn x Nn on L?'{Nn) is a multiplicity free 
direct integral of the irreducible unitary representations vr„,t. 

Remark 6.10 From the considerations just described, one sees that Lemma 16.91 holds for every 
2-step nilpotent Lie group that has square integrable representations. 

We now continue the argument as in the Heisenberg group case. Suppose that the index m ^ n. 
Then \Piibn,t)\^^'^ fi,m,t ^ |Pf (bm,t)|^''^/i,m,t defines an isometric injection ^n,ipit) ^ ^m,y^it) of £n,t 
into £m,t- The norm computation just above gives 



(6.11) 



l^n^,|Pf(^,„,)/Pf(^>™.)l^/Vlli^(iV„) = I (E,,^KPf(^".*)/Pf(^™.*)l \Vhm{t)\') \PKb„,t)\dt 
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Thus we have an {Nn x A'^ft)-equivariant isometric injection 

(6.12) C,n ■■ L^Nn) ^ L\Nm) defined by C,n(*n,^) = '^m,\Pfib„,,)/Piib^.,)\^/^^- 

On the level of coefficients it is given by C'm,ni^n,t,^) = ^m,t,|Pf(b„,t)/Pf(bm,t)|i/V- '^"^^^ 
C^^„ sends the function J2\,ni^lm{t)\Pi{K,t)\^^'^ f\,in;t on to the function on iV^ given by 

l,m l,m 

As in the Heisenberg case we now have 

Theorem 6.13 There is a strict direct system {L'^ {Nn) , Cm n) of L'^ spaces. The direct system 
maps C'mn • Li'^i^n) ^ L'^i^m) {^n X Nn)-equivariant unitary injections. Let Hn denote the 
left/right regular representation of Nn x Nn on L'^{Nn) and let N = lini Nn . Then we have a well 
defined Hilhert space L'^(^N^ : — Ihii^L'^ (^Nn) , Cm n\ ^ natural unitary representation 11 — lirnll^ 
of N X N on L'^{N). Further, that representation U is multiplicity-free. 

Remark 6.14 Exactly as in Remark 13.6^ the adjoint of : L'^{Nn) L'^{Nm) is orthogonal 
projection, and on each {Nm x A'^m)~irreducible direct integrand of Lp'^Nm) it is a scalar multiple 
of restriction of functions. <C> 

7 Structural Preliminaries 

In this section and the next we work out some structural results for a strict direct system {K^ '■Pm,n} 
of compact connected Lie groups and a consistent family {7n} of representations of the Kn on a 
fixed finite dimensional vector space 3. In Section[8]we will use that information to extend Theorems 
15.61 and lS.lOl to a larger family of strict direct systems of nilmanifold Gelfand pairs. 

As just indicated, {Kn., ^m,n} is a strict direct system of compact connected Lie groups. Denote 
K = luii{Kn,fm,n}- Its Lie algebra is t = lim{6„, dc^^.n}- The 7^ : Kn — > U{^) are unitary 
representations of the Kn on the finite dimensional vector space 3. They are consistent in the sense 
that 7m • fm,n = 7nj SO they define the direct limit representation 7 = lim7„ of K on 3. 

Let Un = 7n(-ft^n)- The C/„ form an increasing sequence of compact connected subgroups of 
dimension ^ (dim 3)^ in the unitary group U{}), so from some index on they are all the same 
compact connected subgroup U of U{}). Truncating the index set we may assume that every 
^n{Kn) = U , in particular that, in the limit, U = ^{K). Let k}i denote the identity component of 
the kernel of 7„. Then ipm,n{Kn SO we have — lini { Kn , (pm n \ jy-t }■> Sind is the identity 

component of the kernel of 7. In particular its Lie algebra 6^ = limjtJi, (i(/9^ ^Lj } is the kernel of 
d'y -.1 ^ u. 

Since Kn is compact and connected, and Kn is a closed connected normal subgroup, Kn has 
another closed connected normal subgroup L„ such that Kn is locally isomorphic to the direct 
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product Kn X Ln- On the Lie algebra level, in = K.(Bin, direct sum of ideals. The semisimple part, 
'n = is the direct sum of all the simple ideals of in that are not contained in tn. Thus 1'^ is 

independent of the choice of L„, and dipm,n{^n) = ^m- 

Proposition 7.1 One can choose the groups Ln so that fm.niLn) = Lm for m ^ n » 0. 

Proof. We argue by induction on r = dim(u/[u, u]). If r = then the l„ = so dipm,n{^n) = 
says dipm,niD = U, and it follows that ipm,niLn) = Lm- 

The group U = ^{K) is compact and connected, and the identity component of its center is a 
torus T of dimension r. Suppose r > let let S be a subtorus of dimension r — 1 in T. Now define 
codimension 1 subgroups 'Kn = ^~^{S) C Kn and 'K = ^~^{S) C K. Since 7m • ^m,n = In we 
have ipm,n{'Kn) C 'Km, and 'K = lim{'A'„, ipm,n\'Kn}- By induction on r we have closed connected 
normal subgroups 'L„ C 'Kn such that 'Kn isomorphic to K^ x 'L„ and (pm,n{'Ln) = 'Lm- That 
gives us 'L = lim 'L„ and '[ = lim '[„; 'K is locally isomorphic to K^ x 'L and 't = (B '(. 

Let tt)^ denote the center of and 'tr the center of '[. Let Xo denote the centralizer of 't in 
i. Then tr is an abelian ideal in t that contains ro^ © 'tr as a subalgebra of codimension 1. We 
choose a 1-dimensional subalgebra "to C to not contained in tr^© 'tv and such that the 1-parameter 
subgroup "W = expC'ro) is closed in L. Define [ = "ro© '[. The corresponding analytic subgroup L 
is a closed subgroup of K. For n sufficiently large, say n ^ uq, the direct limit maps ipn ■ Kn ^ K 
satisfy "W C ipn{Kn)- As indicated above, the induction hypothesis gives us 'L C ipn{Kn)- Thus 
L C (pniKn) for n ^ uq. As the (pn ■ Kn ^ K are injective we how have well defined closed 
connected subgroups L„ = {p~^{L) for n ^ no, and 7„ : L„ — > C/ is surjective with finite kernel. 
Thus the (pm,n{Ln) = Lm, and Kn is locally isomorphic to Kn x Ln, for m ^ n ^ uq. That 
completes the proof of Proposition 17.11 □ 

For convenience of formulation we again truncate the index set, this time so that L„ = ip^^{L) 
for all indices n. 

Corollary 7.2 Let t £ and let Kn^t its stabilizer in Kn- If one of the direct systems 
{Kn,(pm,n}, {Kn,t, 'Pm,n\Kn A ^ {Kn, ^m,n\ j^t} is paraboUc, then tkc Other two also are paraboUc. 

Proof. Let Lf denote the stabilizer of t in L. Up to local isomorphism, 

{Kn,(Pm,n} = {K^ X L,ipm,n\f^i X 1} and {Kn,t, V'm,n\K„ t} = {^1 X Lt,ipm,n\f^t X 1}. 

In each case, the direct system is parabolic if and only if {Kn, cpm n\ j^i I is parabolic. □ 
Now Theorem 14.31 gives us 

Corollary 7.3 Let t E 3, and let Kn,t be its stabilizer in Kn- Suppose that the direct system 
{Kn,^m,n} is paraboHc. Then there are natural isometric injections Tn.t,x ^ ^m,t,x for m ^ n, 
from the highest weight X representation space of Kn^t to that of Km.t, md corresponding isometric 
injections Cmn'f^ {{^eg Km,t,x) / {'i^S '^n,t,\)Y^'^ f on spaces of coefficient functions. 
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Another immediate consequence of Proposition 17.11 is 

Corollary 7.4 Let t € 3, and let Kn.t be its stabilizer in Kn- Then L := limL„ is compact, 
K = K^L, and K is locally isomorphic to x L. In particular K acts on i as a compact linear 
group and I has a ^{K) -invariant positive definite inner product. 

8 A Class of Commutative Nilmanifolds, I: Group Structure 

In this section and the next, we make use of the results of Sections [6] and [7| in order to extend 
Theorems 15.61 and 15.101 to strict direct systems of Gelfand pairs that satisfy 

(i) each Gn = Nn x Kn semidirect, Nn connected, simply connected, 
nilpotent with square integrable representations and K^, connected, 
(8.1) (ii) the Kn form a parabohc strict direct system, 

(iii) n„ "-^ n„+i maps centers in — }n+i and complements 0^ ^ Dn+i, and 

(iv) for each n the complement ti„ is Ad(-fir„)-invariant. 

We identify each 3„ with 3 := linr g^. Let Kn denote the identity component of the kernel of the 
action of Kn on 3. The image Ad(Er„)|j of that action is a compact connected group of linear 
transformations of 3. Its dimension is bounded because dimj < 00. We may assume that each 
Ad(i^„)|j = U for some compact connected group U of linear transformations of 3. Proposition 
17.11 gives us complementary closed connected normal subgroups L„ C Kn that map isomorphically 
under Kn ^ Kn+i, so each L„ is equal to L := limL„. Thus we have decompositions Kn = Kn ■ L 
and K = K'^ ■ L where Kn is the kernel of the adjoint action of Kn on 3, K = lini Kn , and 
K^ = lini Kn ■ For each n, AdK„ maps L = Ln onto U with finite kernel. 

If t G 3* write Ot for the orbit Ad*(L)(t). Then Ad*(G)(t) = Ad* {Gn){t) = Ad*(i^„)(t) = Ot = 
L/Lt for each n. Since Ad* (Gn) acts on Ot as the compact group L there is an invariant measure 
ut derived from Haar measure on L; we normalize I't to total mass 1. Given t G 3* we have its 
stabilizers Gt = {g £ G \ Ad* {g)t = t} and Lt = K Ci Gt, and their pullbacks Gn,t and Ln,t- 

Let T = {t G 3* I each Pf(6„ t) / 0}, as in Sectional and fix t G T. As in Section [S] the square 
integrable representation TTn^t extends to a unitary representation vr^ of Gn,t ■= Afn x Kn^t on the 
same representation space Hn^t- If ^n,t,A G Kn^t has representation space !Fn,t,x we write for 
its extension to a representation of Gn,t on J-n,t,\ that annihilates Nn- Then we have the irreducible 
unitary representation vr^^ ^ := vr^ ® k^^\ of Gn,t on 7^^^ := 'Hn,t ® ^n,t,\- That gives us the 
unitary representation TTn,t,\ = I^d^" ^ (vr^^ ^) of G„. Its representation space 

^n,t,A := / (^n,Ad*(fc)t ® ^n,t,x) dl^tiKt)) 
JOt 
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consists of all measurable functions (p : Gn — > ^ \ such that 

(i) ip{gm) = vr,^^ ^{'m)^^ip{£) for g G G„ and m G Gn,t and 

(ii) / ||^(<7)|P<iz.t(Ad*(5)(t))<oo. 
JOt 

In other words t A is the space L^{Ot;M.^^j^) of L2 sections of the homo geneous Hilbert space 
bundle ^n,t,\ ~^ ^^^h fiber W^^y The action TTn,t,x of G„ on ?^„,^,A is [{'^n,t,x{9)){^)]{g') = 
^{g'^g')- The inner product on nn,t,x is (y',^)^ , , = foA^(.9),'>P{9))nO 'ii'tiM* {g){t)). 

According to the Mackey little group theory, (i) -7Tn,t,\ is irreducible, (ii) ■7Tn,t,\ is equivalent to 
'^n,t',x' if and only if t' G Oj, say t' = Ad* {i)t where i £ L, and Ad*(£) carries A to A', and (iii) 
Plancherel-almost-all irreducible unitary representations of G„ are of the form TTn^t,x where t G T 
and K.n,t,x e Kn,t- 

Corollary 17.21 tells us that the system {Kn^t} is parabolic. As noted in Corollarv l7.3l that gives us 
isometric injections J^n,t,x ^ ^m,t,x of representation spaces and corresponding isometric injections 
of the spaces of coefficient functions. Those injections combine with the corresponding maps of 
SectionElto give us isometric injections Cm,n ■ f ^((|Pf(&m,i)| deg Km,f,A)/(|Pf (&n,t)| deg Kn,t,x)Y^'^ f 
from the space of coefficient functions of vr^^ ^ to that of vr^ ^ ^. Those come out of unitary injections 
'^ntx '^mtx °^ representation spaces. The representation space injections define unitary 
(on each fiber) injections ^ ^ t x corresponding homogeneous Hilbert space bundles 

over the orbit Ot- Spaces of sections correspond by 

L\Ot;Ml^^,) =Wg L\Ot;Mt^^,^^) \ ^(^(t)) e ^(Wj,,) for all i e L}. 

The inner products on 'Hn,t,x = ^'^{'^t'l^^t x) '^m,t,x = -^^(^'t; ^ _^) are Gn-invariant, 

and Gn is irreducible on L'^(Ot\ ^), so there is a real scalar Cm,n > such that i— > Cm,nP gives 
a G„-equivariant isometric injection of 'Hn,t,x into 'Hm,t,x- Summarizing to this point. 

Proposition 8.2 As just described we have a strict direct system {TCn.t,x} based on Gn-equivariant 
isometric injections 'Hn,t,x ^ T~(-m,t,x- 

Now consider the spaces Sn,t,x ■= T~(-n,t,x ^^ntA' Hilbert space completion of the space of 
coefficient functions f^^^ : g ^ {p,'^n,t,x{g)'^)n for G T~(-n,t,x- The (G„ x G„)-invariant 
inner product on £n,t,x is 

(8-3) ^ V', v^' ^ V'').^,,,, = ^(^, (Wv:: 

for some number dn^t.x > 0, which we interpret as the formal degree deg7rn^t,x- See Appendix A, 
specifically Theorem A.l below, for a discussion of this. In any case, the right/left action of G„ x G„ 
on £n,t,x is an irreducible unitary representation, and the isometric embeddings 'Hn,t,x ^ T~(-m,t,x 
define isometric embeddings Cm,n : / ^ {deg7rn+i,t,x/ degTrn,t,x)^^'^ f of £n,t,x into £m,t,x- That gives 
us 
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Proposition 8.4 As just described we have a strict direct system {£n,t,\T Cm,n} , whose spaces are 
the Hilbert spaces of coefficients of the representations TTn,t,x, <ind whose maps are {Gn x Gn)- 
equivariant isometric embeddings. 



The group K acts on 3* through its compact subgroup L, so 3* has an Ad*(-R')-invariant inner 
product. Let S be the unit sphere. Since L and S are compact there are only finitely many orbit 
types Lgj^, ■ ■ ■ , of L on 5 ( |Yan| . or see [P] Theorem 1.7.25]). In other words every isotropy 
subgroup of L on 5 is conjugate to exactly one of the Lg-. Thus every isotropy subgroup of K on 
S is conjugate to exactly one of the Ks^ = K'^Ls^. If t G 3* and r 7^ then the isotropy groups 
Kt = Krt- Now every isotropy subgroup of K on j* \ {0} is conjugate to exactly one of the Kg.. 

Decompose S Ci T = Si U ■ ■ ■ U Sm where Si is the union of the orbits Ad* {K){s) in 5 n T 
with isotropy Kg = Kg^. Locally Si contains a smooth section to the action of K. Thus there is 
a measurable section Ui : K\Si — > Si to the action of such that each isotropy K^^aiix) = Kn,si- 
Let Sj = ai{K\Si). According to the Mackey little group theory, Plancherel almost all irreducible 
representations of Gn are of the form ■Kn,t,x where t = rsi with 1 ^ z ^ m and r > and with 
«^n,t,A G Kn,t- That gives 



™ /"CO /■ 

^5) L\Gn) = y] V / / £n,rs,xdsdr 

— - — Jr=0 Js£T,i 



As n increases we have the isometric equivariant injections Cm.n '■ £n,t,x — *■ £m,t,x of Proposition 18.41 
When we form the discrete and continuous sums of (j8.5p . the Cm.n act on the summands, where 
they fit together to define isometric equivariant injections (which we also denote Cm.n) from L?'{Gn) 
to L^{Gm), m^n. That yields the first assertion of 



Theorem 8.6 Let {{Gn,Kn)} be a strict direct system of commutative pairs that satisfy (18. ip . 
Then the isometric equivariant injections £n,t,x — > £m,t,x of Proposition [8^ define {Gn x Gn)~ 
equivariant isometric injections Cm.n '■ L'^{Gn) — > L^{Gm)- That gives a direct system {L^(Gn), Cm,n} 
of Hilbert spaces and equivariant isometric injections. Let n„ denote the left/right regular repre- 
sentation of Gn X Gn on L'^{Gn) and let G = lirn Gn ■ Then we have a well defined Hilbert space 
L'^(G) := lim{L^(G„), Cm,n} ^'^'^ 0, natural unitary representation H = lirn n„ of G x G on L^{G). 
Further, that representation 11 is multiplicity-free. 



9 A Class of Commutative Nilmanifolds, II: Manifold Structure 

We now pass from L'^{G) to L'^{G/K) for strict direct systems of commutative spaces that satisfy 
(jS.ip . Retain the notation of Section [HI The first step is 

Theorem 9.1 Lett G T. Then {NnXKn^t-,Kn,t) is a G elf and pair. In particular Kn^t is multiplicity 
free on C[t)n]. 

Note the similarity between the statement of Theorem 19.11 and Yakimova's commutativity criterion 
f jY3l Theorem 1], or see [WH Theorem 15.1.1]). 
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Proof. We may assume that t = Si, representing one of the orbit types of L on S n T. Suppose 
that {Nn XI Kn^t, Kn^t) is not a Gelfand pair. Then the commuting algebra A for the representation 
Xt of Nn XI Kn^t on L'^{{Nn X Kn^t)/Kn^t) is not commutative. Let ^1,^2 £ A. with A1A2 / 
A2A1. Note that L'^{{Nn x Kn)/Kn) is the sum (over j) of the representation spaces of the 
A,- := Ind ['!^"^!^"-' n(As,). In other words it is the sum of the spaces L^((A^„ x KA/Kn) where 

Lj{{Nn X Kn)/Kn) consists of the L^{Kn/Kn,t) functions (/? : x K„ ^ L^((iV„ x Kn,Sj) / Kn,Sj) 
such that ip{gh) = Xs.{h)^^ip{g) for g G x Kn and /i G Nn x Kn^sy Now define ^1 and by 
=^Au{^{g)). Then M^){gh) = Au{v{gh)) = Au[\s,{h)-\{g)) = \s^{h)-\Au{^{g))) = 
Xs^{h)~^ {Auif){g) , so Au is a well defined linear transformation of L'j{{Nn x Kn)/Kn)- Further, 
[Z,ig) ■ AMKgi) = {M'p)){g-^gi) = AMg-^gi)) = Au{[>^^{g)ip]{gi)) = K(A^.(5)v')](9i), 
so Au is an intertwining operator for A^^ . As the A^ do not commute, neither do the A^. Since 
(A^„ X Kn,Kn) is a Gelfand pair this is a contradiction. We conclude that {Nn x Kn,t, Kn,t) is a 
Gelfand pair. In particular, now, Kn,t is multiplicity free on C[X)n] by Carcano's Theorem. □ 

Recall the Hilbert bundle model for the induced representation TTn,t,\ G Gn,t given by TTn,t,\ = 
Indg" ^(tt^j ^). The representation space Hn,t,x of TTn,t,\ consists of all L'^{Kn/Kn,t) sections of the 
homogeneous bundle p : ^ntx ~^ Gn/Gn,t = Kn/Kn,t whose typical fiber is the representation 
space Ti.^^ ^ of vr^^ ^. Given k G Kn we write k • H'^^ ^ for the fiber p~^{kKn,t)- Let u G be 

^ '''"n t aC-^".*)"^-'^^'^ vector. Then u belongs to the fiber 1 • Ti.^^ ^, and k-u ^ k- Ti-nt \ depends 
only on the coset kKn^t- Define a section 

(9.2) au ■■ Kn/Kn,t ^ ^"i^t^x by au{kKn,t) = k ■ u. 

Then (t„ is a 7r„^(^A(-f^n)^iivariant unit vector in the Hilbert space 'Hn,t,x- (We will also write (pu 
for the corresponding function G„, — > ^ such that ^u{ggt) = tt^^ a(5*)~^(''^"(5')) ^'^^ 9 ^ ^'^'^ 
gt G Gn,t-) Conversely if a is a 7r„^f^A(-f^n)~iiivariant unit vector in 'Hn,t,Xi then a{lKn,t) = c« where 
|c| = 1 by /C„^t-invariance, and then a = ca^ by X-invariance. In summary. 

Lemma 9.3 Let t £ T and let u be the unique (up to scalar multiple) tt^^ ^{Kn,t) "fixed unit vector 
in 'H^^)^ Then the section cr„, given by (j9.2p . is the unique (up to scalar multiple) 'Kn,t,x{K) -fixed 
unit vector in 'Hn,t,x- 

By Theorem 19. II we can apply Proposition 18.41 to the function spaces Sntx — "^n t x ^ (^n tx)* 
on the groups Gn,t = Nn x Kn,t- Now combining Proposition 18.41 and Lemma 19.31 we have 

Proposition 9.4 If orthogonal projection £n+i t x ~^ ^nt x ^^^ds a nonzero right t -mwariant 

function to a nonzero right Kn^t 'invariant function, then orthogonal projection £n+i,t,x ~^ ^n,t,x 
sends a nonzero right Kn+iAnvariant function to a nonzero right KnAnvariant function. 

Vinberg ( |Vlj . [V2j : or see |W2l Table 13.4.1]) classified the maximal irreducible nilpotent 
Gelfand pairs. A Gelfand pair {Gn,Kn) is called maximal if it is not obtained from another Gelfand 
pair {Gn,Kn) by the construction {Gn,Kn) = {G'n/C,Kn/{Kn H C)) for any nontrivial closed 
connected central subgroup C of G'n- And {Gn,Kn) is called irreducible if Ad{Kn) is irreducible 
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on On, = n„/3. Here is Vinberg's classification of maximal irreducible nilpotent Gelfand pairs; see 
[W2) for the notation. 



(9.5) 



Maximal Irreducible Nilpotent Gelfand Pairs (A^„ x i^„,i^„) ([VlJ, [V2j) 




Group Kn 


On 


3 


(7(1) is 
nnerlerl if 


max 
requires 


1 


SOin) 


M" 


SkewM"^" = 5o(n) 






2 


Spinil) 


M« = 


W = Im 






3 




W = Im 


M'^ = Im0 






4 


U{1) ■ SOin) 


C" 


ImC 




n / 4 


5 


(U(l)-)SU(n) 


C" 


A^C"©ImC 


n odd 




6 


SU(n),n odd 


C" 








7 


SU{n),n odd 


C" 


ImC 






8 


Uin) 


C" 


ImC"^" =u(n) 






9 


{U{iy)Sp{n) 


M" 


Reli;}''" elmM 






10 


U{n) 




M 






11 


{U{l)-)SU{n),n ^ 3 




M 


n even 




12 


U(l) ■ Spinil) 


C« 








13 


U{1) ■ Spin{9) 




M 






14 


(U(l)-)Spin(W) 




M 






15 


U{1) ■ G2 


C'^ 


M 






16 


U{1) ■ Ee 




M 






17 


Sp{l) X Sp{n) 


w 


ImE[ = sp(l) 




n ^ 2 


18 


Sp{2) X Sp{n) 




ImH^^^ =5p(2) 






19 


{U{l)-)SU{m) X SU{n) 




R 


m = n 




20 


{U{1)-)SU{2) X SU{n) 




ImC^^^ = u(2) 


n = 2 




21 


{U{iy)Sp{2) X SUin) 




M 


n ^ 4 


n ^ 3 


22 


U{2) X Sp{n) 




ImC^^^ = u(2) 






23 


C/(3) X Sp{n) 




M 




n ^ 2 



As noted in [W2], in Table [93] one often can replace Kn by a smaller group in such a way 
that {Gn,Kn) continues to be a Gelfand pair. For example, in Table [931 Item 2, where A^^^ is the 
octonionic Heisenberg group H01, the pairs (A^^^ x Spin{7), Spin{7)), {Nn x Spin{Q), Spin{6)) and 
(A''„ X Spin{5), Spin{5)) all are Gelfand pairs; see [Ll Proposition 5.6]. This is the tip of the iceberg 
for the classification of commutative nilmanifolds. A systematic analysis is given in |Y2j : or see 
[W2l Chapter 15]. 

The strict direct systems in Table [93] with dimj^ bounded, are as follows. Here we split entry 
line 4 of Table 19.51 so that {Kn} is parabolic, and we split entry 20 into two essentially different 
cases. 
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(9.6) 



Direct Systems of Maximal Irreducible Nilpotent Gelfand Pairs {N„, x Kn,Kn) 




Group Kn 


On 




nonHpH if 


max 
rcQuircs 


4a 


U(l) ■ SO{2n) 




ImC 




n / 2 


4b 


U{1) ■ SO{2n + l) 




ImC 






7 


SU{n),n odd 


C" 


ImC 






10 


U{n) 




R 






11 


{U{l)-)SU{n),n^ 3 




R 


even 




17 


Sp{l) X Sp{n) 




ImH = sp(l) 




n ^ 2 


18 


Sp{2) X Sp{n) 




ImH^^^ =sp(2) 






19 


(U{l)-)SU{m) X SUin) 




R 


m — n 




20a 


SU{2) X SU{n),n^ 3 




ImC"^" =u(2) 






20b 


(7(2) X SU{n) 




ImC^^^ = u(2) 






21 


((/(l)-)5p(2) X SU{n) 


(8)C" 


R 


n ^ 4 


n ^ 3 


22 


(7(2) X Sp{n) 




ImC'" =u(2) 






23 


!7(3) X Sp{n) 




R 




n ^ 2 



In each case of Table [921 tW2' Theorem 14.4.3] says that Nn has square integrable representations. 
In the cases dim 3 > 1 of Table 19.61 we have = K' ■ K'^ where the big factor K'^ acts trivially on 
3 and the small factor K' acts on 3 by its adjoint representation. Summarizing these observations, 

Proposition 9.7 Each oj the thirteen direct systems {{Gn, Kn)} 0/ Table 19.61 /las the properties (i) 
{Kn} is parabolic (ii) the {Kn^Si} parabolic and (iii) A'^^ has square integrable representations. 

The following result is immediate from the multiplicity free part of Theorem 19.11 and the argu- 
ment of Lemma 15.71 

Corollary 9.8 Let {{Gn, Kn)} be one of the thirteen direct systems Table \9M and let t £ T. Let 
Kn,\ e Kn,t- Define G A''™ x Kn,t by K^\{h, k) = Kn,t,\{k). Then ir^^^j^ ■=^t(^ /^Q^ has a 

nonzero Kn,t~fixed vector if and only if k* ^ ^ occurs as a subrepresentation of7L^t\K„ t> <ind in that 
case the space of Kn,t-fixed vectors has dimension 1. 

Corollarv 19.81 lets us apply the argument of Corollarv 15.91 to the representation spaces Ti.^^ ^ := 

T~(-n,t®^n,t,\ of the vr^^ ^. If m ^ n it shows that every X„^(-invariant vector in is the image of 

a Km,t-invariant vector under the adjoint of the unitary map "H^^ — > Ti.^ ^ ^. Combining this with 

Proposition 18. 41 we see that orthogonal projection £^ ^ ^ — > ^ sends nonzero right i^m,t~iiivariant 

functions to nonzero right K„^f-invariant functions. (Here recall the space ^ := Ti.^^ x^C^tt a)* 
of functions on Gn,t-) Now Proposition 19.41 gives us 

Proposition 9.9 Let {{Gn, Kn)} be one of the thirteen direct systems Table WM Let t £ T and 
m ^ n. Then orthogonal projection £m,t,\ ~^ £n,t,\ sends nonzero right Km-invariant invariant 
functions to nonzero right Kn -invariant functions. 

Combining Theorem 18.61 with Corollarv 19.81 and Proposition 19.91 we arrive at 
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Theorem 9.10 Let {{GmKn)} he one of the thirteen direct systems of Table 19.61 Denote G = 
lim Gn and K = lim/C„. Then the unitary direct system of Theorem 18.61 restricts to a 

unitary direct system {L^(G„/i^„)}, the Hilhert space L'^{G/K) := limL^(G„/X„) is the subspace 
of L^{G) := hm L?{Gn) consisting of right-K -invariant functions, and the natural unitary repre- 
sentation of G on LP'{G/K) is a multiplicity free direct integral of lim-irreducible representations. 

Now we go past the cases that require irreducibihty of Kn on d„. 

In the Table 19.141 below, denotes the generalized Heisenberg algebra Im F + F" of real 
dimension 1 + n dim^ F where F denotes the complex number field C, the quaternion algebra H, or 
the octonion algebra 0. It is the Lie algebra of the generalized Heisenberg group Hn-w given by 

-ffn,F '■ real vector space ImF + F" with group composition 

(z, w){z' ,w') = {z + z' + lm h{w, w'),w + w') 

where h is the standard positive definite hermitian form on F". The generalized Heisenberg groups 
Hn-^ all have square integrable representations }jW2\ Theorem 14.3.1]. 

In Table 19.141 we have direct sum decompositions 

(9.12) n = n' © 3" where n' has center 3' = [n, n] and n has center 3' © 3" = [n, n] © 3". 
and i^-stable vector space decompositions 

(9.13) n = 3 + and n' = 3' + D = [n, n] + d. 

Finally, in Table 19.141 su(n) does not mean the Lie algebra, but simply denotes its underlying 
vector space, the space of n x n skew-hermitian complex matrices, as a module for Ad(C/(n)) or 
Ad(5C/(n)). 

Here is a small reformulation of Yakimova's classification of indecomposable, principal, maximal 
and S'p(l)-saturated commutative pairs (A^ x K,K), where the action of K on n/[n,n] is reducible. 
Compare }jW2\ Table 13.4.4]. See |Y2| or |W2j for the technical definitions; for our purposes it 
suffices to note that these are the basic building blocks for the complete classification described in 
[Y2] and [W2j . We omit the case [n, n] = 0, where = M" and K is any closed subgroup of the 
orthogonal group 0(n). 
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(9.14) 



Maximal Indecomposable Principal Saturated Nilpotent Gelfand Pairs (A^ xi K,K), 
N Nonabelian Nilpotent, Where the Action of K on n/[n, n] is Reducible 





Group 


i^—module D 


module 3' [n, n] 


module 3'' 


Algebra n' 


1 


U(n) 




R 


5u(n) 


fln;C 


Z 


TT( A\ 




ini 83 A 


M 


ImC + A^C + C* 


6 


U \ i) X U [ri) 


!L (±) i V 


TD) ID) 

M tr) -K 


U 


fln;C © fl,i(n-l)/2;C 


A 

4 


QT T { A\ 


!L = M 


im *L tp He M 


M 


ImC + Reff^^ + C* 


r 



(7 (2j X U (4j 


X 4 


im (L 


TT3) 


ime^^ + e^* 


D 


D\U (4j X U [m) ) 






M 


fl4m;C 




U [m ) X u [n) 


^ y3 ^ 


TO) /T\ U 
M tt) -K 


U 


[)mn;C © flm;C 


5 


U[i) X op^rij X U[i) 




TD) rr\ ID) 

K tr) -K 


U 


f)2n;C © f)2n;C 


9 


Sp{l) X X U{1) 




Ime©]R 





f)n 


H © l)2n;C 


10 


X bp[n) X op(l) 


M M 


Im M © im H 







H © f|n;H 


11 


Sp(n) X {Sp{l),U{l),{l}} 
xbp{m) 


e" 


ImH 


jjj-jT, X m 


[)n 


H 


12 


Sp(n) X {S'p(l),[/(1),{1}} 


e" 




ImM 


f)n;H 


13 


Spm(7) X {S'0(2),{1}} 


R« = 


R' =Im0 


JJYX2 


f)l 





14 


(7(1) X Spin{7) 


C 


R 


R" 


f)7 


C 


15 


(7(1) X S'pm(7) 




R 


R'' 


t)8 


c 


16 


(7(1) X [7(1) X Spin{8) 


c+ ©cl 


R © R 





f)8 


C © f)8;C 


17 


U(l) X 5'pin(10) 




R 


— mTm 

R^" 


fll6;C 


18 


{SC/(n),t7(n),;7(l)5p(^)} 
xS'C/(2) 




R 


su(2) 


fl2n;C 


19 


{S[7(n), C7(n), t7(l)S'p( 2)} 
xC/(2) 


c © c 


im TTTl 

R © R 





fl2n;C © f)2;C 


20 


{SC/(n),C7(n),;7(l)5p(f)} 

xS[/(2)x 
{SC/(m),f7(m),[/(l)5p(f )} 




R©R 





f)2n;C © fl2m;C 


21 


{SC/(n),C7(n),;7(l)5p(f)} 
xS'l7(2) X f7(4) 




R©R 


R'' 


fl2n;C © f)8;C 


22 


(7(4) X (7(2) 


(^4X2 


R 


R" ©su(2) 


f)8 


C 


23 


(7(4) X (7(2) X (7(4) 


^4X2 ^^2X4 


R©R 


R" ©R" 


[)8 


C © l)8;C 


24 


(7(1) X (7(1) X 5(7(4) 




R©R 


R" 


f)4 


C © f)4;C 


25 


((7(1).)5(7(4)(.50(2)) 




]jbX2 


R 


[)4 


C 



In each case of Table 19.141 the group N = N' x Z" has square integrable representations }jW2\ 
Theorem 14.3.1]. In fact, if t € 3* we decompose t = t' + t" where t'{i") = = t"{i'), and then 
Pf(6t) = Pf(6t'), independent of t" . 

The strict direct systems in Table [9T^ with dim 3^ bounded, are as follows. Here the index i 
can be n or (m, re), the group Gi = Ni x Ki, and the subgroup G'^ = N'^ yj Ki. 
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(9.15) 

Strict Direct Systems {{Ge,Ke)} and {{G'e,Ki)} of Gelfand Pairs From Table 1014] witF dimSf Bounded 





Group Ki 


_ftr£-module 0^ 


module 3^ = [tif, nj>] 


module 3" 


Algebra 


1 


U{n) 


C" 


R 


su{n) 


f|n;C 


3 


(7(1) X U{n) 


C" © A^C" 


R©R 





fln;C © l)n(n-l)/2;C 


6 


S{U{4) X U{m)) 




R 


R'' 


fl4m;C 


7 


U{m) X U{n) 




R©R 





i}mn;C © t)m;C 


8 


C7(l) X Sp{n) X (7(1) 


C^" © C^" 


R©R 





i}2n;C © fl2n;C 


9 


Sp{l) X Sp{n) X (7(1) 


H" ©H" 


ImHffiR 







a © i)2n;C 


10 


5'p(l) X Sp{n) X Sp{l) 


H" ©H" 


ImMffilmH 







a © l)n;H 


11a 


Sp{n) X Sp{l) X Sp{m) 


H" 


ImH 


X Tn 


f|n;I 


a 


lib 


Sp{n) X (7(1) X Sp{m) 


H" 


ImH 


UJTt X Tfl 




a 


11c 


Sp{n) X {1} X Sp{m) 


M" 


ImM 


JJJTl X TTl 


f)n;I 


a 


18a 


SU{n) X SU{2) 




R 


su(2) 


fl2ri;C 


18b 


U{n) X 5(7(2) 


^^Tt X 


R 


su(2) 


fl2ri;C 


18c 


(7(l)5p(f) X 5(7(2) 




R 


su(2) 


fl2n;C 


19a 


5(7(n) X (7(2) 


C"^^ © 


R©R 





fl2n 


;C © f)2 


c 


19b 


(7(n) X (7(2) 




R©R 





fl2n 


;C © f)2 


c 


19c 


(/(l)5p(f ) X (7(2) 


C"'''' ©C^ 


R©R 





fl2n 


;C © i)2 


c 


20aa 


5(7(n) X 5(7(2) x 5f7(m) 




R©R 





i)2n 


;C © [)2m 


c 


20ab 


SU{n) X 5(7(2) X (7(m) 


j^TlX^ f^'ZXTIT, 


R©R 





i)2n 


;C © f)2m 


c 


20ac 


5(7(7i) X 5(7(2) X (7(l)5p(f ) 




R©R 





fl2n 


;C © fl2m 


c 


20ba 


(7(n) X 5(7(2) x 5(7(m) 


^^Tt X *^ X TfL 


R©R 





f|2n 


;C © i)2m 


c 


20bb 


(7(n) X 5(7(2) x (7(m) 


l^nX'Z jj-^liXm 


R©R 





f|2n 


;C © f|2m 


c 


20bc 


(7(n) X 5(7(2) x (7(l)5p(f ) 




R©R 





i)2n 


;C © [)2m 


c 


20ca 


(7(l)5p(5) X 5(7(2) X 5(7(m) 


ij^nXli (j-iL^Xm 


R©R 





i)2n 


;C © i)2m 


c 


20cb 


(7(l)5p(f ) X 5(7(2) X U{m) 


jj-^nXli (^'ZXm 


R©R 





i)2n 


;C © f)2m 


c 


20cc 


(7(l)5p(f) X 5(7(2)x 

U{l)Sp{^)} 


^nx-^ ^£2Xm 


R©R 





i)2n 


;C © i}2m 


c 


21a 


SU{n) X 5(7(2) X (7(4) 


£UX2^£2X4 


R©R 


R'' 


fl2n;C © fl8 


c 


21b 


(7(n) X 5(7(2) x (7(4) 


^nX2 ^^2XA 


R©R 


R'^ 


f|2n;C © f)8 


c 


21c 


(7(l)5p(f ) X 5(7(2) X (7(4) 


(^nX2 ^f^2Xi 


R©R 


R'' 


i)2n;C © f)8 


c 



By inspection of each row of the table we arrive at 

Proposition 9.16 Each of the 28 strict direct systems {{G'^^, Ke)} of Table 19. 151 satisfies ()8.ip . 

If i ^ k in the index set then 3^ ^ 3^ is surjective. We identify each of the 3^ with 3' := lim3^. 
As in Lemma [6^ we have a^, the zero set of the polynomial Ff{bi^t') on 3^, and 0' := |J is a set of 
measure zero in (3')*. And as in (|6.5|) we denote T' = {f G (3')* | each Pi{be^t') / 0} = (3')* \ a'. 

If t' € T' then i^^^j', G'^^, = X: K'^^i and G^^j/ = Ni yi K^^t' are its respective stabilizers in K^, 
G'^ and G£. Theorem 19.11 tells us that {G'^ ^, , Ki^t') is a Gelfand pair, so in particular the action of 
Kg^t' on C[X)i] is multiplicity free. As Gg^t' and G'^^, correspond to the same multiplicity free action 
of Ke^t' on C[d£], Carcano's Theorem says that {Gg^t' , Ki^t') is a Gelfand pair. Now Lemma 19.31 and 
Proposition 19.41 apply. As above, this leads to 

Theorem 9.17 Let {(G^,i^^)} be one of the twenty eight direct systems of Table 19.151 Denote 
G' = limG^ and K = limK'^. Then the unitary direct system {L^(G^),C^~} given by that of 
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Theorem 18.61 restricts to a unitary direct system the Hilbert space L?'{G' /K) := 

liui \L'^{G'f/K£), C,'^ ~| is the subspace of Lp'{G') := lim \ L'^ (Gp) , C'^ -} consisting of right-K -invariant 

functions, and the natural unitary representation of G' on Lp'iG' /K) is a multiplicity free direct 
integral of lim-irreducible representations. 

Now define 3" = lini 3" and let Z" := lim Z'/ denote the corresponding vector group. In Table 
I9.15l the 3^' are constant except for the entries of row 1, where 3" = su(cxd), and rows lla,b,c, where 
3" can be to any of EI"><~, H'^^'" and EI°°^°°. In any case, (3")* = lim(3")*- 

In the cases where the 3" are zero, i.e. 3" = 0, we have Gi = G'^, so the natural unitary 
representation of G = lini Gj on Lp'^G/K) is multiplicity free by Theorem 19.171 

The cases where the 3" are nonzero but constant are 



(9.18) 





Group Ke 


A'^-module Vi 


module i\ = [ni,ni\ 


module 3" 


Algebra 


6 


S'(t/(4) X U{m)) 




R 


R" 


f)4m;C 


18a 


SU{n) X 5(7(2) 


(^n X Z 


R 


su(2) 


f)2n;C 


18b 


U{n) X SU{2) 


(^n X Z 


R 


su(2) 




18c 


f/(l)S'p(f ) X SU{2) 




R 


su(2) 


f)2n 


c 


21a 


SU{n) X SU{2) X U{4) 




R©R 


R" 


f)2n 


c e f)8 


c 


21b 


U{n) X SU{2) X 17(4) 




R©R 


R" 


f)2n 


c e f)8 


c 


21c 


;7(l)S'p(f ) X 5'!7(2) X f/(4) 




R©R 


R" 


f)2n 


c e 1)8 


c 



In those cases {{Gf^Kf)} satisfies (|8.ip . so the considerations leading to Theorem 19.171 applv 
directly to the {((?£, K()}. We turn now to the other cases. They are given by 





Group Ki 


Kt-vaoAule Ve 


module 3^ = [n^, n^] 


module 


Algebra tif 


1 


U{n) 


C" 


R 


su(n) 




11a 


Spin) X Sp(l) X Sp{fn) 


e" 


ImH 


j^n X m 


f)n;H 


lib 


Spin) X (7(1) X Sp(m) 


e" 


ImH 


j^n X m 


f)n;H 


He 


Spin) X {1} X Spim) 


e" 


ImH 


j^n X m 


f)n;H 



For entry 1 in Table EH] we have T' = (3')* \ {0} = E \ {0}. The generic orbits of Ke = U{n) 
on 3^' = su(n) are those for which all eigenvalues of the matrix in su(n) are distinct, so the generic 
t = {t',t") £ T are those for which the stabilizer Ki f is a maximal torus in Kg. Here note that 
the irreducible subspaces for Kg^t on C[\}£] = C[C"] each consists of the multiples of a monomial, 
so the action of Ki^t on C[t)f] is multiplicity free. Now, exactly as in the considerations leading up 
to Theorem 19. 101 the natural unitary representation of G on Lp'{G/K) is a multiplicity free direct 
integral of lim-irreducible representations. 

The argument is more or less the same for entries 11a, b,c in Table [9T9l Here T' = (3')* \ {0} = 
ImH \ {0} where ImH is identified with its real dual using the inner product (z^, Z2) = Re (z^Zg). 
The action of Ki = Sp{n) x {Sp{\),U x Sp{m) on = H" is {ki,k2,h) : v ^ kivk2, 
on 3^ = ImH is {ki,k2,ks) : z' \—>- k2z'k2, and on 3" = M"^™ is {ki,k2,ks) : z" ^ kiz"k^. 
Here the composition H" x H"" ImH is {v,w) 1— > lmv*w and k2 is a quaternionic scalar. The 
stabilizer of t' G T' in Kg is Sp{n) x {f/(l), U{1) or {±1}, {1}} x Sp{m); generically that stabilizer 
is Sp{n) X {J7(l), {±1}, {1}} X Sp{m). Again, as in the considerations leading up to Proposition 
19.91 and Theorem 19.101 the natural unitary representation of G on L?'{G/K) is a multiplicity free 
direct integral of lim-irreducible representations. In summary. 



26 



Theorem 9.20 Let {{Gi, K£)} be one of the twenty eight direct systems of Table I9.15[ Then the 
unitary direct system {L^(G^),Q^} analogous to that of Theorem 18.61 restricts to a unitary direct 

system {Lp'{Gi/ K(),C,£j\ ■ The Hilbert space L?'{G/K) := lim{L^(G£/i^^), Q j} is the subspace of 
L'^{G) := lun{L'^ (G i) , Ci (} consisting of right-K -invariant functions, and the natural unitary rep- 
resentation of G on LP'{G/K) is a multiplicity free direct integral of lim-irreducible representations. 



Appendix A: Formal Degrees of Induced Representations 

In this section we work out the formal degree of an irreducible induced representation lnd^,j{-j), 
where 7 is a square integrable representation of L, L/M is compact, and L/AI has a positive L- 
invariant measure. The result, which is suggested by (|8.3p . is not surprising, but does not seem to 
be in the literature. 

Let L be a separable locally compact group, M a closed subgroup of L, and J a closed central 
subgroup of M that is normal in L. Suppose that 7 is an irreducible square integrable (modulo 
J) unitary representation of M. Then 7 has well defined formal degree deg7 in the usual sense: 
if u, V, u' and v' belong to the representation space Ti^, and if we write fu,v for the coefficient 

fu,v{^) = {u,l{fn)v)n-,, then {fu,v, fu'y) l^m/j) = d^(^i, ^^'X^^i ^')- 

The modular functions Am and Al coincide on M. This is just another way of saying that we 
have a unique (up to scale) L-invariant Radon measure d{iM) on L/M, and thus an L-invariant 
integral / ^ /^/^^^ f{e)d{eM) for functions / G Gc{L/M). 

Denote 7 = Ind^j(7) and let be its representation space. The elements of Ti-y are the 
measurable functions tp : L ^ Ti^ such that (i) ip{£m) = j{m)~'^ip{i) (for £ G L and m € M) and 
(ii) f^/f^^\\ip{e)\\'^d{m) < 00. The action 7 of L on T^y is [^{£)(p]{£') = ifii'^t). The inner product 
on 71, is (<^,V')^ = J^^^,{ip{e),m)n,dm)- 

The group M acts on the fiber TC^y of TC^y L/M at IM by 7, and more generally the stabilizer 
iMi~^ of the fiber iTC^ at £M acts on that fiber by ^£{imi~^){(.v) = I ■ ^{m)v. Since J is normal 
in L it sits in iM£~^ and 7^ | j consists of scalar transformations of £Tly. Now the representations 

all are square integrable mod J and have the same formal degree deg7. 

If u,v £ Ti-y we have the coefficient function fu,v{n^) = (^i7("^)^)7i^ ™ V^i^ ^ "^7 

we have the coefficient ftp^ii) = {(p,^{i)'ip)~ on L. Now let f,ip £ Tiy such that, as functions 
from L to 7i^, both ip and -0 are continuous. Their support is compact modulo M because L/M 
is compact. Now f^^^ : L ^ C has support that is compact modulo M. For every i £ L, 



ftp,ip\iM is a matrix coefficient of and f^y\eM 



2 



has integral (integrate over IMi ^ / J) equal to 



2 \\,I,(P\\\2 



The functions I ^ \\vU)\\^ and I 1— > ||v^(^)||?, are continuous on L/M, 
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so their L?'{L/M) inner product converges. Now 

/ \U,^{l)\''d{m) = [ ( [ \U,4im)\''dimJ)] dim) 
JL/J JL/M \Jm J 

Thus 7 has a nonzero square integrable (mod J) coefficient. Since it is irreducible, all its coefficients 
are square integrable (mod J). In particular the formal degree deg7 is defined. In summary, 

Theorem A.l Let L he a separable locally compact group, M C L a closed subgroup, and J a 
central subgroup of M that is normal in L. Suppose that L/M is compact and has a nonzero L- 
invariant Radon measure. Let ^ he a square integrable (modulo J) irreducible unitary representation 
of M and deg7 its formal degree such that 7 := Ind^(7) is irreducible. Then 7 is square integrable 
(modulo J) and it has a well defined formal degree deg7. 

Of course, if L/M is finite, then Theorem A.l becomes trivial, and there if we use counting 
measure on L/M then deg7 = |L/M|deg7. However, in effect we use the result in (j8.3p where 
L/M is compact but infinite. 



Appendix B: A Computational Argument For Theorem 19.11 

In this appendix we give a computational proof of Theorem 19.11 for the spaces of Table 19.61 This 
provides somewhat more information and could be useful in studying the spherical functions. In 
order to align the X^-invariants in L^(G„) and pass to L^(G„/i^„) we will need 

Theorem B.l Let {{Gn, L^n)} be one of the thirteen direct systems Table [9^ and let t £T. Then 
the representation of Kn^t on C[i3„] is multiplicity free. 

Proof. If dim 3 = 1 the assertion follows from Carcano's Theorem. That leaves table entries 17, 
18, 20a, 20b and 22. In those cases we explicitly decompose C[dn] under the Kn^t- 

In the case of entry 17, the big factor K!/ = Sp{n) is irreducible on 0„ = = C^", and 
Kn,t = U{1) X Sp{n). The representation of Kn^t on C[gt'„] is the same as that of the Gelfand pair 
listed on row 4 of Table 15.11 and also row 4 of Table 15.21 It follows that the representation of Kn^t 
on C[Vn] is multiplicity free. 

In the case of entry 20a we have Kn^t = K'^ ^ x SU{n) where K'^ ^ is the circle group consisting 
of all ka = (^0 i/a) with \a\ = 1. Then d„ = C" ©C" where ka acts on C!^ as multiplication by o^^. 
Define (i) Pn,i,mi is the space of polynomials of degree mi on C!J:, (ii) Pn,i,m2 is the space of polyno- 
mials of degree m2 on C" , and (iii) Pn,m = Pn,i,mi ©IF'n,i,m2 where m = (mi, m2). Then Kn^t acts on 

mi mi — m2 m2 
™n,i,mi by X o o • • • o and on Pn,2,m2 by x o o • • • o . Thus 

the representation of Kn,t on P„,m is Y,p+2q=mi+m2\^ ^ "^^ ^ ° ° ° )• 
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Multiplicities in the representation of Kn,t on C[t)n] would lead to equations m'l — m'2 = nii — m2, 

p' = p and q' = q with p' + 2q' = m'l + 777,2 ^^'^ p + 2q = mi + m2, forcing 777'^ = mi and 7773 = 777,2- 
That is a contradiction because representation of Kn,t on m multiplicity free. We conclude 
that the representation of Kn^t on C[d„] is multiplicity free. 

In the case of entry 20b, the representation of K^^t on C[d„] is multiplicity free as a consequence 
of the multiplicity free result for entry 20a. 

In the case of entry 22 we view 0„ as eg) C^". Here Kn,t = -f^n,* x 5*^(71) where K'^ ^ is the 
2-torus group consisting of all ka,b = (ob) with \a\ = \b\ = 1. Then 0„ = C" © C" where ka,b 
acts on C" as multiplication by a and on C" as multiplication by b. The representation of Sp{n) 

on either of C" is o o . . . o < o , so the representation on polynomials of degree d 

is the symmetric power S"^( o o . . . o <. o ) = o o . . . o < o . Now the 

branching rule (with r ^ s) 

r s 

O O • • • r-' O O • • • 0=^^=0 

7 S V 

^ ( o c c ) 

v=0 u=0 

shows that the representation of Kn,t on Pn,m is 

|mi-m2| V 

/ mi 7712 7711 + 7712 — 2v U ^ 

2^ (. X X o o ° «^ ). 

v=0 u=0 

If two such irreducible summands are equivalent, say for {mi,m2,u,v) and {m'l^m^iu' ,v'), then 
evidently mi = m[, 7772 = mi + r772 — 27; = m'^ + 77^2 — 2v' and u = u'. We conclude that the 
representation of Kn,t on C[t)„] is multiplicity free. 

In the case of entry 18 the Kn t correspond to the centralizcrs of tori (of dimensions 0, 1 or 2) 

' a 13 

in Sp{2). If we view 5*^(2) from its diagram o < o the centralizers Kn^t = -^n,t ^ Sp{n) are 
given up to conjugacy by K'^^^ = Sp{2), K'^^^ = U{2) with simple root a, K'^ ^ = U{2) with simple 
root (3, and K'^ ^ = maximal torus of -5^(2). Specifically, if we realize sp(2) as the space of 2 x 2 
quaternionic matrices ^ with ^ + ^* = 0, it has basis 

( i 0\ (jO\ (kO\ (0 0\ (0 0\ (0 0\ ( 1\ (0 i\ (0 j\ i / o fc ^ 

\oo)^ \oo)^ \oo)' \oi)^ \oj)^ Kok)' l-ioJ' UoJ' \j 0) \ko)- 

Write the complexifying C as M + 6IR where 6^ = — 1 , to avoid the notation of the quaternions used 
to express sp(2), and let £1,82 be the usual linear functional (that pick out the diagonal entries of 
a matrix) . Using the Cartan subalgebra spanned by ( q ) ^^'^ ( ? ) ' 

{■'^^ q) is a root vector for it ei and (q is a root vector for it €2', and 

( ^^'^^ ) is a root vector for ± {ei — £2) and {j^bk ''^^ ) ^ ^^^^ vector for it (ei + £2)- 

Thus the centralizer of Za ( ° ) is the direct sum of the Cartan subalgebra with the span of 
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(if ai = / as): ( ^0 ° ) and ( g ), so the centrahzer of Za is ( " ) ® ( o « 

(if ai / = as): ( [J ° ) and ( « ), so the centrahzer of Za is ( o sp(i) ) © ( * o ); 

(if 7^ ai 7^ as / 0): the centrahzer of Za is the Cartan subalgebra, span of ( q [] ) and ( [J ^ ); 

(if ai = as 7^ 0): ( i^i q ) ^ii^ ( ? o )' centrahzer of Za is u(2) as 2 x 2 complex matrices; 

(if ai = —as 7^ 0): ( j q) ^'^'^ (fc o)' ^° centrahzer of Za is isomorphic to u(2). 

Here the first two cases are 5p(2)-conjugate, and the last two cases are 5p(2)-conjugate, because 
the Weyl group consists of all signed permutations of the e^. Thus we really only have three 
cases up to S'p(2)-conjugacy. The nilpotent group Nn is the (very) generalized Heisenberg group 
-f^2,n,0;H of |W2t Section 14.3], so |W21 Theorem 14.3.1] shows that Za corresponds to a square 
integrable representation of Nn precisely when ai 7^ 7^ as. Now we need only consider the cases 
(i) ai = as 7^ and (ii) 7^ ai 7^ as 7^ 0. 

Suppose ai = as 7^ 0. Then the corresponding Kn,t = SU{2) x Sp{n). Since dim„ On — 
dirajjEI^^" = 8n we are now viewing D as C'^^^", and the representation of Kn^t (as a subgroup 
of U{n)) on 0„ is the irreducible {k',k") : v 1-^ k'v{k")~^. As (A^„ x Kn,t,Kn,t) is a Gelfand pair, 
Carcano's Theorem says that the representation of Kn^t on C[dn] is multiplicity free. 

Suppose 7^ ai 7^ as 7^ 0. Then Kn,t = Uil) x U{1) x Sp{n) and we view as C2>^2n ^ cf"©Ci" 
where the action r of is r(A;^, ^2; ^'Ol'^i © ^2) = {k[vi{k")~^ k2V2{k")~^). The representation 
r has diagram ( x x ° — o — ° — • • • — ) © ( x x o — o — o — . . • — c=^» ) . Now decom- 
position of the symmetric powers S'^{t) goes exactly as in the case of entry 22, and we conclude 
that the representation of Kn^t on C[t)„] is multiplicity free. 

We have shown, by direct computation for each of the series of Table [9^ that the representation 
of Kn^t on C[l3n] IS multiplicity free. □ 

One can carry out similar calculations for the series of Table [9TT5] showing by direct computation 
that in each case the representation of Kn,t on C[o„] is multiplicity free. 
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